Skip to content

[HUDI-5585][flink]Fix flink creates and writes the table, the spark alter table reports an error#7706

Merged
danny0405 merged 2 commits intoapache:masterfrom
waywtdcc:flink_write_spark_alter_error
Feb 1, 2023
Merged

[HUDI-5585][flink]Fix flink creates and writes the table, the spark alter table reports an error#7706
danny0405 merged 2 commits intoapache:masterfrom
waywtdcc:flink_write_spark_alter_error

Conversation

@waywtdcc
Copy link
Copy Markdown
Contributor

@waywtdcc waywtdcc commented Jan 19, 2023 •
edited
Loading

Change Logs

Fix flink creates and writes the table, the spark alter table reports an error

After the flink hive catalog is created, it does not include meta information fields in the hive metadata, such as _hoodie_commit_time, etc. However, these fields are included in the spark creation table hive. So it leads to metadata incompatibility issues.

Impact

Fix flink creates and writes the table, the spark alter table reports an error

Risk level (write none, low medium or high below)

low

Documentation Update

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@danny0405 danny0405 self-assigned this Jan 20, 2023
@danny0405 danny0405 added flink-sql area:engine-interop Engine interoperability priority:medium Moderate impact; usability gaps labels Jan 20, 2023
@danny0405
Copy link
Copy Markdown
Contributor

danny0405 commented Jan 20, 2023

Thanks for the fix @waywtdcc , can we describe in high level what we are fixing here?

@waywtdcc
Copy link
Copy Markdown
Contributor Author

waywtdcc commented Jan 31, 2023

Thanks for the fix @waywtdcc , can we describe in high level what we are fixing here?

This is spark and flink metadata incompatibility issue. The table created by flink cannot be changed by spark.

danny0405
danny0405 reviewed Jan 31, 2023
View reviewed changes
...ink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/catalog/HoodieHiveCatalog.java Outdated
schema = HiveSchemaUtils.convertTableSchema(hiveTable);
}
org.apache.flink.table.api.Schema resultSchema = DataTypeUtils.dropIfExistsColumns(schema, HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION);

Copy link
Copy Markdown
Contributor

@danny0405 danny0405 Jan 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In line419, we already ignore the metadata column, so why drop it again?

Copy link
Copy Markdown
Contributor Author

@waywtdcc waywtdcc Jan 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, I have removed.

@danny0405
Copy link
Copy Markdown
Contributor

danny0405 commented Jan 31, 2023

HUDI-5585.patch.zip
Thanks for the contribution, I have reviewed and attache a patch, can you apply the path with cmd:

git apply xxx.patch

then rebase with the latest master code and force push?

@danny0405 danny0405 added priority:high Significant impact; potential bugs and removed priority:medium Moderate impact; usability gaps labels Jan 31, 2023
@waywtdcc waywtdcc requested a review from danny0405 January 31, 2023 05:49
danny0405
danny0405 approved these changes Jan 31, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@danny0405 danny0405 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@hudi-bot
Copy link
Copy Markdown
Collaborator

hudi-bot commented Jan 31, 2023

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@danny0405
Copy link
Copy Markdown
Contributor

danny0405 commented Feb 1, 2023

The two failed tests are un-related with this patch:

https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=14803&view=logs&j=3b6e910d-b98f-5de6-b9cb-1e5ff571f5de&t=30b5aae4-0ea0-5566-42d0-febf71a7061a&l=421297

https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=14803&view=logs&j=7601efb9-4019-552e-11ba-eb31b66593b2&t=9688f101-287d-53f4-2a80-87202516f5d0&l=7247

Would merge it soon~

@danny0405 danny0405 merged commit 9469882 into apache:master Feb 1, 2023
yihua pushed a commit that referenced this pull request Feb 2, 2023
@waywtdcc @danny0405 @yihua
[HUDI-5585][flink] Fix flink creates and writes the table, the spark …
22b634b 
…alter table reports an error (#7706)


Co-authored-by: danny0405 <yuzhao.cyz@gmail.com>
@danny0405 danny0405 added priority:blocker Production down; release blocker and removed priority:high Significant impact; potential bugs labels Feb 5, 2023
@chenbodeng719
Copy link
Copy Markdown

chenbodeng719 commented Mar 16, 2023 •
edited
Loading

Which released version includes this feature? Hudi 0.12 version has to stick with this problem?

@danny0405
Copy link
Copy Markdown
Contributor

danny0405 commented Mar 16, 2023

0.13.0

nsivabalan pushed a commit to nsivabalan/hudi that referenced this pull request Mar 22, 2023
@waywtdcc @danny0405 @nsivabalan
[HUDI-5585][flink] Fix flink creates and writes the table, the spark …
c986ba4 
…alter table reports an error (apache#7706)


Co-authored-by: danny0405 <yuzhao.cyz@gmail.com>
fengjian428 pushed a commit to fengjian428/hudi that referenced this pull request Apr 5, 2023
@waywtdcc @danny0405
[HUDI-5585][flink] Fix flink creates and writes the table, the spark …
9030c29 
…alter table reports an error (apache#7706)


Co-authored-by: danny0405 <yuzhao.cyz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@danny0405 danny0405 danny0405 approved these changes

Assignees

@danny0405 danny0405

Labels

area:engine-interop Engine interoperability priority:blocker Production down; release blocker

Projects

Hudi PR Support
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@waywtdcc @danny0405 @hudi-bot @chenbodeng719